Statistical Language Modelling

Grammar-based natural language processing has reached a level where it can ‘understand’ language to a limited degree in restricted domains. For example, it is possible to parse textual material very accurately and assign semantic relations to parts of sentences. An alternative approach originates from the work of Shannon over half a century ago [41], [42]. This approach assigns probabilities to linguistic events, where mathematical models are used to represent statistical knowledge. Once models are built, we decide which event is more likely than the others according to their probabilities. Although statistical methods currently use a very impoverished representation of speech and language (typically finite state), it is possible to train the underlying models from large amounts of data. Importantly, such statistical approaches often produce useful results. Statistical approaches seem especially well-suited to spoken language which is often spontaneous or conversational and not readily amenable to standard grammarbased approaches. This chapter concerns statistical language modelling. In a speech recognition system the role of the language model is to assign probabilities to word sequences. Recently, similar models to speech recognition language models have been employed to perform higher level tasks, such as structuring and extracting information from spoken language. In this chapter, we first outline the basic framework of n-gram language models (section 2), which form the core of current statistical approaches. A crucial technical consideration here is how to estimate n-gram statistics from sparse training data. We go on to describe two approaches—based on n-gram models—to encapsulate varying contents and styles: section 3 is concerned with mixture language models and section 4 builds on the observation that the occurrence rate of a word is not uniform, but varies between documents. Finally we describe a statistical finite state model for the extraction of information, such as proper names and dates from spoken language.